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DETAILED ACTION 

Response to Arguments 
1. Applicant's arguments with respect to claims 1, 8, 9, 10 5 16, 17, 23, 24, and 28-29 filed 
09/09/2005 have been considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1-3, 5-6, 8-11, 16-19, 22-24, and 30-34, are rejected under 35 U.S.C. 102(e) as 
being anticipated by Schofield et al. (US 6.498,620 B2) in view of Kumamoto Kenji (JP 10- 
257482). 

Re claims 1, 8-10, 16-17, 23-24, and 30-34, Schofield discloses an image processing 
apparatus comprising an image processing part (18 of fig. 5) operable to receive images captured 
by a plurality of cameras (14 and 16 of fig. 7) shooting surroundings of a vehicle (12 of fig. 1) 
and operable to generate a synthetic image (col. 5, lines 59-64) viewed from a virtual point of 
view (col. 6, lines 1-4) from these camera images (col. 5, line 65-col. 6, line 4), wherein the 
image processing part (18 of fig. 5) changes at least one selected from a position, a direction of a 
line of sight, and a focal length of the virtual point of view in accordance with a running state of 
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the vehicle ( col. 21, line 41-col. 22, line 6; note the image processor (18) responds to the 
temporal and spatial patterns of infrared signals detected by image capture devices (cameras 14, 
16) in order to determine (selected) the speed and distance and, thereby, the separation of the 
vehicles as well as the rate of change of separation of the vehicles as considered a running state 
of the vehicle). 

Moreover, Scho field teaches the image processing part (18 of fig. 22) changes at least 
one selected from a position (col. 21, lines 25-41, e.g. the image processing changes a position in 
accordance with a running state (moving) of the vehicle), a direction of a line of sight (col, 21, 
line 67-col. 22, line 6,. e.g. changing a field of view from rearward to forward of, or to the side 
of, the vehicle; see also col. 14, lines 1-29), and a focal length of the virtual point of view in 
accordance with a running state of the vehicle (cameras 14 and 16 of fig. 24, e.g. the cameras 14 
and 16 obviously have a focal length so when the vehicle changes from one location to other 
location the focal length of the cameras also change); and controlling capturing of an image 
outside a view range (cameras 14 and 16 of fig. 4, e.g. the cameras 14 and 16 captures the 
images outside a view range of the driver when the vehicle moving forward direction as consider 
running state, and the captured images can be displayed on the display 20' of fig. 10) of the 
virtual image in accordance with a running state of the vehicle. 

Additionally, Schofield teaches the image processor (18 of fig. 5) generates an image 
including a first image viewed from the virtual point of view (14 Left of fig. 5) and a second 
image viewed from a different viewpoint and a different model (14 Right of fig. 5; see also 14A; 
and 19 of fig. 5; e.g. the camera 14A Left captures the fist image at different angle from the 
camera 14A Right that captures the second image, the cameras 14A Right and Left are have 



Application/Control Number: 09/807,654 Page 4 

Art Unit: 2613 

different viewpoint from each other); and wherein in the synthetic image, the image processing 
part (20 of figs. 3 and 5) displays at least a part of a vehicle region (42 of fig. 3) where the 
vehicle is present, and an attention drawing region (50 and 52 of fig. 3) for drawing attention in 
which at least a part of the surroundings of the vehicle is shown; wherein the attention drawing 
region includes at least a part of a blind spot region around the vehicle that is not shot by any of 
the cameras (20 of fig. 3, note the display (20) is displaying the image that includes a part of 
blind spot, a part of a blind spot regions around the vehicle that is not shot by any of the cameras 
as considered harsh marks (70a and 70b of fig. 6), harsh marks are displayed on the display 20" 
to indicate the blind spot drawing regions; see also col. 17, lines 10-64). 

It is noted that Schofield suggests the synthesized image would be displayed using other 
display techniques such as to provide a projected or virtual image; alternatively a virtual image 
may be displayed on an opaque display adjacent the forward field of view (col. 8, lines 22-55) 
using the received image from the cameras (14 and 16 of fig. 5). 

However, Schofield does not particularly teach the synthetic image showing the car from 
a virtual point of view that is converted from the actual viewpoint, wherein the virtual point of 
view is set outside and above the vehicle as claimed. 

Kenji teaches the synthetic image (42 of fig. 5-8) showing the car (46 of fig. 5) from a 
virtual point of view (8 of fig. 8) that is converted from the actual viewpoint (6, 10, 11, 14, and 
15 of fig. 2; 37 of fig. 3); wherein the virtual point of view is set outside and above the vehicle 
(42 and 46 of fig. 5; Note the virtual point is outside and also above the vehicle). 

Therefore, taking the teachings of Schofield and Kenji as a whole, it would have been 
obvious to one skill of ordinary skill in the art to incorporate the teachings of Kenji into the 
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image processing apparatus of Scho field for displaying a generated synthetic image showing the 
vehicle from a virtual point of view using the real image captured from cameras. 

Doing so would provide an always clean car circumference image can be seen by 
detecting the variation rate of the car body by the irregularity of a road surface by the 
acceleration sensor, feeding back to a display image, and removing the shake of an image. 

Re claim 2, Scho field further teaches wherein the image processing part (18 of fig. 5) 
changes at least one selected from a position, a direction of a line of sight, and a focal length of 
the virtual point of view in accordance with a running speed of the vehicle (fig. 10 and fig. 11, 
note when the vehicle turns to the left or right, a position is changed so that the image capture 
devices (14 and 16 of fig. 5) picks up a new images based upon the changed position). 

Re claim 3, Schofield further teaches wherein the image processing part (18 of fig. 5) 
changes at least one selected from a position, a direction of a line of sight, and a focal length of 
the virtual point of view, and controls capturing of an image outside a view range of the changed 
virtual point of view (col. 14, lines 1-29, see also fig. 7, note distances R, S, PI and P2). 

Re claim 5, Schofield further teaches wherein the image processing part changes at least 
one selected from a position, a direction of a line of sight, and a focal length of the virtual point 
of view in accordance with a steering angle of the vehicle (XVII of fig. 12, see also col. 17, lines 
31-36). 

Re claim 6, Schofield further teaches wherein the vehicle includes an object detection 
sensor (176 of fig. 21, note the object sensor (176) may be a distance-measuring device, such 
scan active infrared sensor, an ultrasonic sensor, a radar sensor, or the like. Such object sensor is 
especially useful in determining the separation distance between the vehicle and objects in front 
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of the vehicle. Preferably, object sensor (176) has a sensing field of view that is substantially 
coincident with the field of view of one or more of the image capture devices 14, 16) for 
detecting an obstacle, and the image processing part changes at least one selected from a 
position, a direction of a line of sight, and a focal length of the virtual point of view in 
accordance with results of detection by the object detecting sensor (176 of fig. 21). 

Re claims 10 and 16, Schofield further teaches an image processing apparatus comprising 
an image processing part (18 of fig. 21) for receiving images captured by a plurality of cameras 
(14 and 16 of fig. 2 1) shooting surroundings of a vehicle to generate a synthetic image from 
these camera images, wherein the image processing part (18 of fig. 21) generates an image 
including a first image (the captured image is from the camera 14 or 16) as the synthetic image, 
the second image being viewed fig. 3, the right image being viewed on the display is different 
from the left image) from a viewpoint that is different from the virtual point of view of the first 
image in at least one selected from a position, a direction of a line of sight and a focal length, or 
the second image being different from the first image in a model. 

Re claim 1 1 , Schofield further teaches wherein the second image is at least one of the 
camera images (fig. 3). 

Re claim 18, Schofield further teaches wherein the synthetic image is an image viewed 
from a virtual point of view that is set above the vehicle (fig. 3). 

Re claim 19, Schofield further teaches wherein the image processing part displays an 
illustration image or an actual image of the vehicle on the vehicle region (20 of fig. 3, see also 
col. 7, line 47-col. 8, line 12). 
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Re claim 22, Schofield further teaches wherein the image processing part determines a 
range of a region obtained by combining the blind spot region and the vehicle region, using 
region data showing a projection region of the vehicle in each camera image (76 of fig. 14., see 
also col. 9, lines 7-1 1). 

3. Claims 4, 11-15, and 26-27 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Schofield et al. (US 6.498,620 B2) in view of Kumamoto Kenji (JP 10-257482) in view of 
Shimizu (US 5,796,991). 

Re claims 4, 1 1-15, and 26-27, Schofield teaches an image processing apparatus 
comprising an image processing part (18 of fig. 5) for receiving images captured by a plurality of 
cameras (14 and 16 of fig. 5) shooting surroundings of a vehicle to generate a synthetic image 
from these camera images, wherein the image processing part (18 of fig. 5) generates the 
synthetic image, using a mapping table (col. 14, lines 30-56) including first mapping data 
describing a correspondence relationship between pixels of the synthetic image and pixels of the 
camera images (Left image and Right image are from the outputs of the cameras 14 in figure 5, 
see also fig. 3), and second mapping data describing an identifier showing that a pixel of the 
synthetic image corresponds to pixel data other than the camera images (58 of fig. 3); wherein 
the pixel data other than the camera images show the vehicle or a blind spot region that is present 
in at least a part of the surroundings of the vehicle (70A and 708 of fig. 6). 

It is noted that the combination of Schofield and Kenji does not particularly teach the 
image processing part stores a predetermined image other than the camera images, and with 
respect to the pixel of the synthetic image, the second mapping data describe coordinate values 
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corresponding to the pixel in the stored predetermined image; wherein the first image is a close 
view image showing the vehicle and surroundings thereof, and the second image is a distant view 
image showing an area distant from the surrounding area of the vehicle that is shown by the 
close view image; wherein the image processing part arranges the distant view image around the 
close view image in the synthetic image; wherein the distant view image is an image having 
continuity with the close view image as claimed. 

However, Shimizu teaches wherein the image processing part stores a predetermined 
image other than the camera images (CG MODEL DATABASE, 244 of fig. 7), and with respect 
to the pixel of the synthetic image (213 of fig. 7), the second mapping data describe coordinate 
values corresponding to the pixel in the stored predetermined image (CG NAGE FORMING 
UNIT, 235 of fig. 7); wherein the second image is at least one of the camera images (20 1L and 
201 R of fig. 7); wherein the first image is a close view image showing the vehicle and 
surroundings thereof, and the second image is a distant view image showing an area distant from 
the surrounding area of the vehicle that is shown by the close view image (fig. 6 A, fig. 10A- 
10E); wherein the image processing part arranges the distant view image around the close view 
image in the synthetic image (figs. 6A and 6B); wherein the distant view image is an image 
having continuity with the close view image (figs. 10A-10E); wherein the first image shows at 
least a part of the vehicle and at least a part of the surroundings of the vehicle, and the second 
image is obtained by enlarging at least a part of the region shown by the first image (figs. 10A- 
10E). 

Taking the combined teachings of Schofield, Kenji and Shimizu as a whole, it would 
have been obvious to one of ordinary skill in the art to incorporate the teachings of Shimizu into 
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the combined image processing part of Schofield and Kenji for the same purpose of synthesizing 
the images with the predetermined model. 

Doing so would provide a natural appearance among images of virtual objects and an 
improved simulation environment. 

4. Claims 28 and 29 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Kumamoto Kenji (JP 10-257482) in view of Shum et al. (US 6,271,847), 

Re claims 28-29, Kenji teaches an image processing apparatus comprising an image 
processing part (fig. 5) for receiving images captured by a plurality of cameras (31-33 of fig. 3) 
shooting surroundings of a vehicle (a car in figure 3) to generate a synthetic image (42 of fig. 5) 
from these camera images, wherein the image processing part uses mapping data (generating) 
images captured by the cameras to form the synthetic image (46 of fig. 5); and the synthetic 
image (42 of fig. 5-8) showing the car (46 of fig. 5) from a virtual point of view (8 of fig. 8) that 
is converted (mapping) from the actual viewpoint (6, 10, 11, 14, and 15 of fig. 2; 37 of fig. 3). 

It is noted that Kenji does not particularly teach describing a correspondence relationship 
between pixels of the synthetic image and a plurality of pixel data including one or both of pixel 
data of the camera images and pixel data other than the camera images, and describing a rate of 
necessity with respect to each of the pixel data, and weights each pixel data in accordance with 
the rate of necessity, thereby generating the pixel data of the pixels of the synthetic image as 
claimed. 

However, Shum teaches describing a correspondence relationship between pixels of the 
synthetic image and a plurality of pixel data including one or both of pixel data of the camera 
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images and pixel data other than the camera images, and describing a rate of necessity with 
respect to each of the pixel data, and weights each pixel data in accordance with the rate of 
necessity, thereby generating the pixel data of the pixels of the synthetic image (fig. 1 1 and 12). 

Therefore, taking the teachings of Kenji and Shum as a whole, it would have been 
obvious to one skill in the art to incorporate the teachings of Shum into the apparatus of Kenji to 
perform the process of mapping weighted pixels and rate. Doing so would provide for better 
blending of images having radically different scales. 

5. Claims 1, 7 and 29 are rejected under 35 U.S.C. 103(a) as being unpatentable over Daily 
et al. US 6,317,127) in view of Kumamoto Kenji (JP 10-257482). 

Re claims 1, 7 and 29, Daily teaches an image processing apparatus (fig. 1) comprising 
an image processing part (48 of fig. 1) for receiving images captured by a plurality of cameras 
(12 of fig. 1) shooting surroundings of airplane to generate a synthetic image (50 of fig. 1) from 
these camera images, wherein the image processing part includes an original mapping table (80 
of fig. 5), cuts out a mapping table describing a correspondence relationship between pixels of 
the synthetic image and pixels of the camera images (figs. 6 and 7), and generates the synthetic 
image, using the cut-out mapping table (108 of fig. 7); wherein the image processing part (40 of 
fig. 1) includes an original mapping table (fig. 5) and generates a synthetic image using a 
mapping table (108 of fig. 7) that is cut out from the original mapping table, and the image 
processing part changes at least one selected from a position, a direction of a line of sight, and a 
focal length of the virtual point of view by changing the mapping table to be cut out from the 
original mapping table (106, 114 and 1 16 of fig. 7). 
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It is noted that Daily does not particularly teach the cameras are mounted on or attached 
to the vehicle to capture the surroundings and the synthetic image showing the car from a virtual 
point of view as claimed. 

However, Kenji teaches the cameras are mounted on or attached to the vehicle to capture 
the surroundings and the synthetic image (42 of fig. 5-8) showing the car (46 of fig. 5) from a 
virtual point of view (8 of fig. 8). 

Therefore, taking the teachings of Daily and Kenji as a whole, it would have been 
obvious to one skill of ordinary skill in the art to incorporate the teachings of Kenji into the 
image processing apparatus of Daily for displaying a generated synthetic image showing the 
vehicle from a virtual point of view using the real image captured from cameras. 

Doing so would provide an always clean car circumference image can be seen by 
detecting the variation rate of the car body by the irregularity of a road surface by the 
acceleration sensor, feeding back to a display image, and removing the shake of an image. 

Conclusion 

6. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
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will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Tung Vo whose telephone number is 571-272-7340. The 
examiner can normally be reached on Monday-Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mehrdad Dastouri can be reached on 571-272-7418. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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